Method and system for handwritten electronic messaging

ABSTRACT

The present invention provides a system and method for providing handwritten messages with electronic communication devices. For example the electronic communication device may be a mobile communication device and may include a touch screen, screen buffer memory, processor, and transceiver. In preferred embodiments, the mobile communication device may be a personal digital assistant (PDA), a two-way pager, or a mobile telephone. The mobile communication device user may create a handwritten message on the touch screen using, for example, a stylus. In one exemplary embodiment, the handwritten message displayed on the touch screen may be stored to the buffer memory as an image, directed by the processor to the transceiver, and transmitted via a wireless connection to, for example, a wireless base station and the Internet. The Internet may route the handwritten message image to a desired Internet address. The handwritten message may be displayed in real time on the recipient&#39;s display, e.g., an instant message (IM) session, or stored in their email mailbox for later retrieval and display. The recipient may type in a text message in response to the hand written message, and this message will be transmitted to mobile communication device and displayed on the mobile communication device. In another embodiment, the receiving party may also have handwritten messaging capabilities (included with, for example, a PC, PDA or a mobile phone) and may respond with a handwritten message.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 11/235,854,filed Sep. 27, 2005, U.S. Pat. No. 7,224,991 which was a continuation ofU.S. patent application Ser. No. 11/009,193, filed Dec. 10, 2004, U.S.Pat. No. 7,003,308 which was a continuation of U.S. patent applicationSer. No. 09/659,653 filed Sep. 12, 2000.

TECHNICAL FIELD

The present invention relates to electronic communications, and moreparticularly, to providing handwritten messaging in electroniccommunications.

BACKGROUND OF THE INVENTION

Presently, a number of electronic communication devices provide textmessaging for a user. A number of mobile communication devices provide2-way text messaging, i.e., the user of the mobile communication devicecan send and receive text messages. Text messages are typewritten usingletters and numbers input via, for example, a keyboard or keypad. Themobile communication device user may type the text message in real timeor may select a message from predetermined canned typewritten messages.For example, the Motorola Pagewriter™ 2000 is a two way text pager thatallows a user to receive a text message email message and return atypewritten text message by typing on a tiny keyboard. As anotherexample, the Motorola V700 word message pager is a two way text pagerwhich allows the user to receive an email message and respond byselecting a predetermined text message previously stored in memory. Attimes, such text messages are difficult to compose or select in realtime because of the user's location or environment. It is oftenconsidered distracting and socially rude to type on a keyboard or fumblewith selection buttons on a pager while, for example, attending ameeting, listening to a lecture, or appearing in a court room. Further,there are times when typing on a small keypad may be impractical; forexample, when a user is riding in a car. Therefore, there is need for aless intrusive and more versatile method of sending messages fromelectronic communication devices, particularly mobile electroniccommunication devices.

SUMMARY OF THE INVENTION

The present invention provides a system and method for providinghandwritten messages with electronic communication devices. According tothe invention, an electronic communication device may include a touchscreen, screen buffer memory, processor, and transceiver. In preferredembodiments, the electronic communication device may be a mobilecommunication device, for example, a personal digital assistant (PDA), atwo-way pager, a mobile telephone, or a simplified wireless touchscreen. The mobile communication device user may create a handwrittenmessage on the touch screen using, for example, a stylus. In oneexemplary embodiment, the handwritten message displayed on the touchscreen may be stored to the buffer memory as a digital image (e.g., agraphic image), directed by the processor to the transceiver, andtransmitted via a wireless connection to, for example, a wireless basestation coupled to the Internet, a PSTN, or a broadband communicationsystem. The handwritten message may also be stored in a system memorythat can accommodate a message which is of greater length then the touchscreen size. The Internet may then route the handwritten message imageto a desired internet address. The handwritten message may be displayedin real time on the recipients display, e.g., an instant message (IM)session, or stored in their email mailbox for later retrieval anddisplay. In one embodiment, the recipient may type in a text message inresponse to the handwritten message, and this message will betransmitted to the mobile communication device and displayed on themobile communication device. In another embodiment, the receiving partymay also have a communication device (mobile or stationary) includinghandwritten messaging capabilities and may respond with a handwrittenmessage. In a further embodiment, the handwritten electroniccommunication message may be automatically re-sized to, for example, thescreen size of the receiving communication device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a first block diagram for the configuration of variouselectrical components which may be included in a mobile handwrittenmessage device in accordance with one embodiment of the presentinvention.

FIG. 1B shows a second block diagram for the configuration of variouselectrical components which may be included in a mobile handwrittenmessage device in accordance with another embodiment of the presentinvention.

FIG. 2 is a process flow diagram illustrating one exemplary method ofproviding handwritten messaging in accordance with the presentinvention.

FIG. 3 shows an example of an electronic communication device includinga touch screen on which a handwritten message may be created inaccordance with one embodiment of the present invention.

FIG. 4 shows an example of the handwritten message shown in FIG. 3 asreceived and viewed on another communication device in accordance withone embodiment of the present invention.

FIG. 5 shows a block diagram for the configuration of various electricalcomponents which may be included in a mobile handwritten message devicein accordance with another embodiment of the present invention.

FIG. 6 shows another example of a hand-held communication device, aPalm™ Handheld Computer (by using a Palm OS™ Emulator) including a touchscreen on which a handwritten message may be created in accordance witha further embodiment of the present invention.

FIG. 7 is a process flow diagram illustrating another exemplary methodof providing handwritten messaging using, for example, a Palm™ HandheldComputer, in accordance with an embodiment of the present invention.

FIG. 8 shows an example of the handwritten message which may be sent inresponse to the message shown in FIG. 6, in accordance with oneembodiment of the present invention.

FIG. 9 shows an exemplary schematic representation of a communicationnetwork according to one embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Today electronic communications are wide spread and include voice andtext communication (e.g., email messaging). Text communications areachieved using a keyboard, keypad, or even a graphic conversion to atyped text (e.g., Graffiti on a Palm™ Handheld Computer). However, thesemethods are cumbersome on small devices and even on larger devices forthose users who are not skilled at typing. Recent development ofelectronics and touch screen displays with high density digital pixelsmake legible handwritten messages possible. Thus, electroniccommunication may now be performed, according to the present invention,using handwritten message images drawn or scribed on a touch screendisplay. Messages created by writing on a touch screen may be moreeasily and quickly created than a typewritten message for most people.Further, handwritten message capability is even more useful whenincorporated in mobile communication devices.

Many busy people carry two way mobile communication devices such asmobile telephones, two-way pagers, personal digital assistants, laptopcomputers, etc., so that they can communicate with others anywhere,anytime. However, presently available manners of communicating withthese mobile communication devices (e.g., typing, talking, etc.) areoften intrusive and fail to provide a manner of unobtrusivelycommunicating at particular times, for example, during a meeting. Thepresent invention provides a system and method for providing unobtrusiveand versatile two-way communications using handwriting.

In the present invention electronic communication devices are equippedfor communications using handwritten messages. In one embodiment, amobile communication device may include a touch screen, processor (e.g.,a microprocessor), and transceiver. This configuration may be augmentedwith a screen memory and/or system memory. In preferred embodiments, themobile communication device may be, for example, a personal digitalassistant (PDA), a two-way pager, a mobile telephone, or simply awireless touch screen. Of course, the communication device need not bemobile but may be of a stationary kind, for example a desktop personalcomputer, including a touch screen or touch pad. The system is providedwith software or firmware which enables the electronic communicationdevice to capture, package and transmit handwritten communications as,for example, an electronic mail message.

In operation, the user of the communication device may create ahandwritten message on a touch screen using, for example, a stylus,finger or other handwriting instrument. Alternatively, the handwrittenimage could be created using a mouse, pointing stick, pen and tablet, orother graphical input device. The handwritten image is retained in amemory of the handwritten communication device, for example, a screenmemory associated with the touch screen device. The message may alsocontain typed text, an HTML-encoded document, images, voice, and/orvideo information. In any case, once created, the system will thentransmit the handwritten message as an image in, for example, an instantmessage (IM) format, email message format such as Simple Mail TransferProtocol (SMTP), or a facsimile format. The SMTP formatted message mayuse a format for including non-text information, for example,Multipurpose Internet Mail Extensions (MIME). Once properly formatted,the message is transmitted to another communication device via, forexample, a wireless communication network and the Internet, where it isdisplayed as an image.

In one exemplary embodiment, the handwritten message displayed on thetouch screen may be stored to a buffer memory as a digital image (e.g.,a graphic image), directed by a processor to a transceiver, andtransmitted via a communication connection (e.g., a wire or wirelessconnection) to, for example, the Internet, a PSTN, or a subscriberbroadband communication system. The communication system, e.g., theInternet, may then route the handwritten message image to a desiredlocation, e.g., Internet address. The handwritten message my bedisplayed in real time on the recipient's display, e.g., an instantmessage (IM) session, or stored in their email mailbox for laterretrieval and display.

The recipient of a handwritten image communication may respond with atyped text message or a handwritten message. In one embodiment, therecipient may type in a text message in response to the handwrittenmessage and this message will be transmitted to the mobile communicationdevice and displayed on the mobile communication device. In anotherembodiment, the receiving party may also have a communication device(mobile or stationary) including handwritten messaging capabilities, andmay respond with a handwritten message.

Further, the handwritten communication device according to the inventionmay scroll the display of the touch screen to accommodate a greaterlength or width handwritten message. This greater length handwrittenmessage may be stored, at least in part, in, for example, a systemmemory large enough to accommodate a message which is of greater sizethen a single screen size. In fact, the screen could include 3D messagecapabilities.

A more detailed description of preferred embodiments of the electronichandwritten communications systems and methods for the present inventionfollows. These embodiments are understood to be merely exemplary and theinvention is not intended to be limited thereto.

A first exemplary handwritten communication device 100 that can provideelectronic handwritten communications is illustrated in FIG. 1A. Thedevice 100 includes a mechanism for displaying incoming communicationsand outgoing handwritten communications, for example, a touch screendisplay 105. The touch screen display 105 is coupled to a databus 120which is used to integrate communications between each component of thedevice 100. The touch screen display 105 may display, for example, anincoming message and/or an outgoing message including a handwrittenelectronic mail message. The system further includes aprocessor/controller 108 and an inter communication system interface 107connected to the databus 120. The processor/controller 108 may be amicroprocessor including memory such as RAM, flash memory, and/or ROMneeded to support the operations of the device. The inter communicationsystem interface 107 may be a wireless communication port or a hard wirecommunication port, including a transceiver, to send and receivecommunications from, for example, the Internet, a PSTN, or a broadbandcommunication system.

As illustrated using dashed lines, the handwritten communication device100 may also include a screen memory 106, an input device 109, an outputdevice 110, a system memory 111, and a character generator 112 coupledto the databus 120. Screen memory 106 may be, for example, RAM or flashmemory and store data indicative of activated or deactivated pixels onthe touch screen display 105 which have been contacted by a usercreating a handwritten communication (or input via a keyboard to createa typed text communication message). Alternatively, this informationcould be stored in the memory of the processor/controller 108. The inputdevice 109 may be, for example, a keyboard or keypad which may bephysically separate, integral to the device, or a simulated keypaddisplayed on a portion of the touch screen display 105. The input devicemay also integrate the touch screen handwriting capabilities previouslydescribed as being related to the touch screen display 105. The outputdevice 110 may be, for example, a speaker, a buzzer, a light, avibrator, or any other mechanism for alerting a user of an incomingmessage. The handwritten communication device 100 may alert a user of anincoming message by flashing all or a portion (e.g., an icon) of thedisplay on the touch screen display 105. The system memory 111 mayinclude, for example, a RAM and/or flash memory and may be used to storea handwritten communication received or to be transmitted by the intercommunication system interface. It may be particularly useful in storinglarge or long messages. The character generator 112 may be a charactergenerator well known in the art which will decode a digitally encodedmessage and generate characters from received communications for displayon the touch screen display (e.g., a text email message).

An alternative system architecture for a handwritten communicationdevice 101 according to the present invention is illustrated in FIG. 1B.In this preferred embodiment, the components of the device are coupledto one another rather than a central databus. Referring to FIG. 1B, thedevice 101 includes a touch screen display 125 coupled to aprocessor/controller 128 and the processor/controller 128 may be coupledto an inter communication system interface. The touch screen display 125may display incoming communications and outgoing communications,including handwritten communications. The touch screen display 125 iscoupled to the processor/controller 128 via signaling line 132. Thesignalling line 132 may carry signals indicating pen touch events on thetouch screen display 125 or may carry signals of an incomingcommunication image. The processor/controller 128 may be, for example, amicroprocessor including memory such as RAM, flash memory, and/or ROMneeded to support the operations of the device. The inter communicationsystem interface 127 may be a wireless communication port or a hard wirecommunication port, including a transceiver, which may send and receivecommunications to and from, for example, the Internet, a PSTN, or abroadband communication system. In a preferred embodiment thetransceiver would have digital capabilities. A screen memory 126 iscoupled to the touch screen display 125 and the processor/controller 128and may provide a store for images, including handwritten communicationsinput by contacting the touch screen display with, for example, astylus. As would be understood by one skilled in the art, the screenmemory 126 may be integral to the touch screen display 125. Thehandwritten communication device 101 may further include a userinterface 129 coupled to the processor/controller 128. The userinterface 129 may be a stylus contacting the touch screen display 125and a sensor for sensing the contact of the stylus. The user interface129 may further include a keyboard or keypad entry device and/or anoutput device such as a speaker. This output device could also be alight, vibrator, etc. Finally, the handwritten communication device 101may include a system memory 131 coupled to the processor/controller 128and a character generator 130 coupled to the processor/controller 128and the touch screen display 125.

The handwritten communication device would further include softwareand/or firmware that enables a user to create a handwritten message bywriting on the touch screen display and transmit that message to anothercommunication device and/or system via the inter communication systeminterface. This process would be supervised by the processor/controller.An exemplary method illustrating the operating of the present inventionincluding appropriate software and/or firmware is provided in FIG. 2.

The handwritten communication device may have multiple communicationmodes including typed text communications, handwritten communications,voice communications, etc. If so, the handwritten communication devicewill contain a means for mode selection such as a button or code enteredvia the touch screen or a keypad. Referring to FIG. 2, once in thehandwriting communication mode, first the handwritten communicationdevice receives handwritten input on the touch screen 205 by a usertouching the screen with a pointing device, for example, a stylus, pen,finger, etc. Then, at decision step 206, the handwritten communicationdevice determines if it is set to the erase mode. If so, then at step207 portions of a previously input image are deleted according to thepixels of the touch screen being touched by the pointing device. If not,then at step 208 the handwritten communication device concludes that theuser intends to create a handwritten communication and stores theidentity of pixels of a handwritten image input via touching the touchscreen display. In either case, the handwritten communication devicethen determines at decision step 209 if the image as created should betransmitted via the inter communication system interface. This may beachieved by activation of, for example, a send button or icon. If theimage is not to be presently sent, the device will continue to allow theimage to be modified by further erasing or handwriting/drawing. If theimage is to be presently sent, then at decision step 210, thehandwritten communication device determines if the communication is partof a real time interactive communication session (e.g., an instantmessage (IM)), part of a delay communication session (e.g., anelectronic mail (email) message), part of a fax message, etc. If thehandwritten communication is to be part of a real time interactivecommunication session, then the method continues to step 213 and thehandwritten communication device sends the handwritten image to anothercommunication device via the inter communication system interface. Forexample, the inter communication system interface may provide wirelesstransmission of the handwritten image through a PSTN and the Internet toan internet address and an active personal computer connected to theinternet address may display the handwritten image in real time. On theother hand, if the handwritten image is being transmitted as part of adelayed communication session such as an email message, then thehandwritten communication device determines at decision step 211 if thehandwritten image should be included with a previously received message.If so, then at step 212 the handwritten image and received message aresent to, for example, an email address and stored in an email directoryas a pending message. The actual handwritten image message may beintegral to the message field or an attachment to the email. In eithercase, the handwritten image message may be accessed and displayed onanother communication device connected to the receiving email addresswith a previously transmitted message. If the handwritten image is notto be included with a previously received message, then at step 214 thehandwritten message itself is sent to another communication device. Ifthe handwritten image is to be sent as a facsimile, then at step 215 thehandwritten image will be sent to a facsimile number.

FIG. 3 provides an illustration of one exemplary touch screen displayand pen or stylus according to the present invention, including anexemplary handwritten communication. The touch screen display 305includes erase 306, receive 307, send 308, and clear 309 buttons. Thesebuttons may be physical buttons or part of a graphical user interface(GUI) activated via touch on the touch screen display. The touch screenmay also have a cursor/pointer 310 which may be used to activate suchbuttons. Further, the touch screen may contain one or more scroll bars311 (vertical and/or horizontal) so that the user can scroll up and downin a message including a handwritten document. The scroll may beactivated with, for example, a stylus or pen 315. The device may alsoprovide auto-scrolling when, for example, the stylus/pen 315 is placedin a predetermined lower portion of the touch screen display area. Forexample, the lower ¼ to ½ inch of the touch screen display could beprogrammed to have a different background color and result in thedisplay automatically scrolling up additional space (e.g., one line)anytime the stylus is placed within this area of the touch screen.

An exemplary handwritten message 320 is provided in the center of thetouch screen display 305. This message is created by a user pressing thestylus of pen 315 on the touch screen display 305 and writing thevarious letters for the words in the sentence. In response the pixelstouched by the stylus or pen 315 turn on and become a contrasting color(e.g., black) to the background color of the display (e.g., white). Asymbol or a drawing (heart shape, check mark, etc.) could likewise becreated as the message or part of the message. Note that the handwrittenimage, or a portion thereof, could be in any color selected by the userfrom a color palette by adding an additional icon and color palettefeature to the electronic handwritten communication device. Further, atyped message or portion of a message could be created if thehandwritten communication device includes a keypad or keyboard feature.To erase a portion of the handwritten image, the erase button 306 may bepressed and the stylus or pen 315 then pressed on the screen over theportion of the image to be erased. If the larger end (erasure) of thestylus or pen 315 is used a larger area will be erased at one time. Inany case, the erasing procedure will turn the color of the pixels whichare touched to the color of the display background (e.g., white).Pressing the clear button 309 will clear the displayed portion of thedocument (i.e., one screen), a selected portion of the screen (i.e.,highlighted portions), and/or the entire document. Pressing the sendbutton 308 will transmit the handwritten communication to the electronicaddress (e.g., internet email address) or telephone number inserted inthe “To” box 312. Pressing the receive button 307 will enable thehandwritten communication device to receive messages. For example, itmight display an email directory or enable a real time interactivecommunication to be received when a user is alerted of the message.

Referring to FIG. 4, an example of the handwritten message shown in FIG.3 as received and viewed on another communication device is provided.The display 305 of another communication device shows displaying thehandwritten image created with the handwritten communication device on adisplay screen 405 of, for example, a personal computer. In this case,the handwritten image was transmitted as an email message but may havebeen sent as an instant message, a fax, or a direct image.Alternatively, the handwritten image could be sent to anotherhandwritten image device (e.g., via infrared (IR) ports). In any case,the handwritten image is displayed on screen in the likeness as it wascreated. However, it may appear on the display 305 of a receivingcommunication device in the same size, a larger size, or a smaller size,than the size at which it was created. The sizing of the image may beprovided by the software or firmware of the handwritten image devicetransmitting the image, by a server within a communication systemtransporting or storing the image, or by the receiving communicationdevice (as will be explained in more detail below). Further, aspreviously noted, the handwritten image may appear in the email messagefield 405 as illustrated in FIG. 4, or as an attachment to the emailmessage format in, for example, attachment field 406.

Referring now to FIG. 5, another embodiment of the handwrittencommunication device is provided. This embodiment is exemplary of amobile handwritten communication device such as a personal digitalassistant (PDA) which may be carried inconspicuously into a meeting. Themobile handwritten communication device 500 includes a touch screendisplay 505 upon which a handwritten communication may be created usinga stylus, pen, or other writing object. A screen memory 506 is coupledto the touch screen display 505 and functions to save the handwrittencommunication image created by a user on the touch screen display 505.As with the previous embodiments, the screen memory 506 may be integralwith, for example, the pixel circuitry of the touch screen display 505.A processor/controller 507 is coupled to the screen memory 506 andcoupled to the touch screen display. The connection 509 between thetouch screen display 505 and the processor/controller 507 allows variouspen events which occur on the touch screen display 505 to be processedby the processor/controller 507 without interacting with screen memory506. For example, selecting a particular mode such as creating and/orsending a handwritten image to another party may be initiated via a penevent sent on line 509 to the processor/controller 507 in response totouching a stylus on a button icon displayed on the touch screen display505. An inter communication system interface 510 may be a transceiver ormodem and is coupled to the processor/controller 507. In a preferredembodiment the inter communication system interface 510 may be awireless interface which operates at radio frequencies and/or aninfrared interface. Finally, the handwritten communication device 500may include a user interface 508 for notifying a user of an incomingmessage.

One exemplary PDA which may use the circuitry illustrated in FIG. 5 isthe Palm™ Handheld Computer. An exemplary embodiment of the inventionwas developed for the Palm™ Handheld Computer using the Palm OS™Emulator as shown in FIG. 6. The Palm OS™ Emulator 605 has been used todevelop and prove out operation of a handwritten communication deviceaccording to the invention by programming it with custom software,identified herein as DoodleMail™ application (Dmail) as illustrated at606. The Dmail software program is based on the Doodle 0.6, releasedApr. 26, 1997, software program developed by Roger E. Critchlow Jr. fordoodling on the Palm™ Handheld Computer and is modified to includeelectronic mail functionality as provided by the present invention. TheDmail 606 software may include the following feature/functions asexplained below.

Referring now to FIG. 6, a Dmail 606 menu bar may include variousfunction buttons such as a draw/erase button (“d” 607), a handwrittenimage/typed text button (“t” 608), a stored message button (“1” 609), areply button (“Reply” 610) and a message waiting button (“Get” 611). Thedraw/erase button 607 functions to toggle between the draw mode (“d”)wherein a handwritten image may be created on the major portion 615 ofthe Palm™ Handheld Computer screen and the erase mode (“e”) whereinportions of a handwritten image displayed on the Palm™ Handheld Computerscreen may be erased (e.g., filled in with the screen background color).The handwritten image/typed text button 608 functions to toggle betweena handwritten image mode (“t”) wherein a user can create a message bywriting on the main portion 615 of the Palm™ Handheld Computer touchscreen with a stylus and the type text mode (“T”) wherein a typed textmessage can be created. A typed text message may be created on a Palm™Handheld Computer using one of two different methods; graffiti ordisplayed keyboard. The default in Dmail may be the graffiti mode. Inthe graffiti mode a typed text message may be created by the userperforming predetermined stylus strokes in the information entry box620. Alternatively, while in the typed text mode (“T”) the user mayactivate a keypad entry method by contacting the stylus in the lowerleft hand corner of the information entry box 620 (where “abcde” isdisplayed) so as to display a touch text keypad on the major portion ofthe screen. The keyboard could also be a physically separate keyboardattached to the Palm™ Handheld Computer via a communication link. Inthese manners, typed text may be included with a handwritten Dmailcommunication. Touching the then displayed “T” button 608 when you aredone entering typed text, returns the device to the handwritten imagemode.

The stored message button 609 (“1”) allows the user to scroll throughall incoming and outgoing messages which are stored until removed. Bytouching the “1” the user may move between messages. Touching the MENUbutton 625, then selecting the Page option, the user will find severalcommands that manipulate stored messages. The “clear” command erases alldrawing and text in the current message. The “new” command creates a newblank message. The “remove all” command deletes all saved messages.Thus, a user can select and manipulate the various incoming and outgoingmessages stored in the Palm™ Handheld Computer. In one variation of theinvention, the user may also select and manipulate the various incomingand outgoing messages stored on a server, for example a handwrittenmessage server, to which the handwritten communication device is coupledvia a communication link.

The “Reply” button 610 allows a user to respond to an incoming message.A user may respond to an incoming message by scribbling on the touchscreen including the incoming message and touch “Reply”. In response,the Palm™ Handheld Computer provides a pop-up window that confirms thesender to whom the user is replying and an “OK” and “Cancel” button (notshown). For example, to reply to the message from Lillian Linnellillustrated on the screen shown in FIG. 6, the user can then enter ahandwritten reply message using Dmail with the draw/erase button 607 andthe handwritten image/typed text button 608 in their “d” and “t” modes,respectively. This handwritten message would be created by scribing onthe main touch screen area 615 with, for example, a stylus.Alternatively, the message could be typed or partially typed andpartially handwritten. Then the user may send the message by activatingthe “Reply” button 610. A pop up window will appear indicating that areply will be sent to Lillian Linnell. Then the user may touch the OKbutton and the handwritten reply message will be sent to LillianLinnell. At this pop up window, the user may enter an alternate oradditional recipient's address.

To get new messages using Dmail, a user may activate the message waitingbutton (“Get” 611) on the Palm™ Handheld Computer. If there are newmessages, the Palm™ Handheld Computer may, for example, beep with thealarm tone and/or flash the screen for thirty seconds. To stop theflashing, the user may touch the touch screen with the stylus. The newmessage is automatically displayed when it is received. If more than onemessage has been received, the user touches the stored message button609 (“1”) to step through the messages.

The user may also program the Dmail to automatically poll for incomingmessages. To set up automatic polling for incoming messages the userwould first touch the menu button 625. Then the user would select the“Message” and “AutoCheck” option features (not shown). Next, the userwould ensure that the “ENABLED” function box is checked (not shown).Then the user would specify how often they wish to check for incomingmessages, e.g., 30 minutes (not shown). If the Palm™ Handheld Computeris powered off with DMail running, it will periodically wake up and pollfor new messages. To stop the automatic polling, power up the Palm™Handheld Computer and start the application launcher 630 or any otherapplication.

This embodiment was developed such that incoming messages must be shortand be limited to a single page including the handwritten reply so as tosimplify the operation to not include a screen scroll feature. In thiscase, the messages will be truncated after the end of a single page.However, one skilled in the art recognizes that a scroll screen featureis well known in the art and may be added along with sufficient memoryallocation for each message so that messages may occupy more than onescreen.

To send a new message a user may first load the Dmail software bytouching the applications button 630, and then clicking on the Dmailicon. Then the user may touch and activate the menu button 625 andselect the “Page” and “New” option features (not shown) to get a blankscreen. Once the main screen area 615 appears blank, the user may scribethe message on the main screen area 615. Next, once the message iscomplete, the user may touch the displayed “Send” button that appears inplace of the “Reply” button when a new message is created (not shown),insert the outgoing address on the pop-up window, and touch the “OK”button.

Prior to using Dmail for the first time, the user will need to configuretheir Palm™ Handheld Computer. First, the user would load the Dmailsoftware onto their Palm™ Handheld Computer. Then the user wouldconfigure the device to properly operate Doodlemail by touching the menubutton 625 and selecting the “Message” and “Configure” option features.The fields available for user input include the SMTP Server, From,Address, POP3 Server, User, and Password fields. The “SMTP Server” fieldidentifies the server for outgoing messages. The “From” field is forinput of the user's name. It will then appear in all outgoing messagescreated using Dmail. The “Address” field is for entering the user'sreturn address (e.g., email address) for messages sent by the user whenusing Dmail. For example, it may be the address to which a person wouldsend the user email at a chosen POP3 server. This address will appear inoutgoing Dmail messages. The “POP3 Server” field is to include theelectronic address of the POP3 server selected by the user. The “User”field would include the user's login name on the POP3 server. The“Password” field is for entering the user's password on the POP3 server.

Referring now to FIG. 7, a process flow diagram illustrating oneexemplary method of operating the handwritten message application using,for example, a PDA such as a Palm™ Handheld Computer is provided. Oncethe handwritten message application, for example Dmail, is activated asa resident application program on the Palm™ Handheld Computer the useris free to operate the application's feature/functions. First, at step705, the program awaits initiation by the user, for example, touchingthe touch screen display 505 with, for example, a pen or stylus. Next,in decision step 706, the system determines whether the pen or styluscontact on the touch screen display 505 is within the handwritten inputdrawing area 615. If so, then at decision step 707, the systemdetermines if the erase mode is activated. If not, then at step 708 thepixels touched by the pen or stylus on the touch screen display 505 areset in, for example, the screen memory buffer (e.g., 506). If the erasemode is set, then at step 709 the pixels touched by the pen or stylus onthe touch screen display 505 are cleared or change to the color of thescreen background in, for example, the screen memory buffer (e.g., 506).This process will continue until the pen or stylus touches an area ofthe touch screen display 505 outside of the drawing area 615.

If the pen or stylus touches a feature/function actuation button, forexample a button displayed on the touch screen display 505 in an areaother then the main drawing area 615, then the program will determinewhat change in program activation is to occur such as those indicativeof steps 710-719. In one instance, at step 710 the program willdetermine whether a message is to be sent (i.e., a new message or areply message). If so, at step 711, the Palm™ Handheld Computer willsend the message stored in, for example, the screen memory 506, to thedesignated electronic address via the inter-communication systeminterface 510. For example, the message may be transmitted via awireless modem to a wireless base station or satellite, via a wire linemodem to a telephone or cable station, or via an infrared transmitter toan infrared receiver (e.g., to another Palm™ Handheld Computer). Inanother instance, at decision step 712, the program will determinewhether a user wishes to check awaiting messages. In this case, the usermay have manually activated the polling of incoming messages by touchingthe “get” button 611. If so, at step 713, the Palm™ Handheld Computerwill poll, for example, the POP3 server, for any incoming messages viathe inter-communication system interface 510. In a further instance, atdecision step 714 the program will determine whether a user wishes toerase the entire drawing area 615 of the touch screen display 505. Forexample, the user may wish to create a new message and touch the menubutton 625 and select the “Page” and “New” options. If so, at step 715,the Palm™ Handheld Computer will clear, or set to the screen backgroundcolor, all pixels included in the drawing area 615 of the touch screendisplay 505.

In a still further instance, at decision step 716, the program willdetermine whether a user has touched the draw/erase button 607 so as toplace the Palm™ Handheld Computer in the erase mode and enable clearingonly certain identified pixels touched by the user. If so, at step 717,the Palm™ Handheld Computer will set the clear mode so that any pixel inthe drawing area 615 of the touch screen display 505 which is touched bythe user will be cleared or set to the screen background color. In aneven further instance, at decision step 718, the program will determinewhether a user has touched the draw/erase button 607 so as to place thePalm™ Handheld Computer in the draw mode so as to enable handwrittenmessage entry by the user touching desired pixels. If so, at step 719,the Palm™ Handheld Computer will set the draw mode and clear the erasemode so that any pixel in the drawing area 615 of the touch screendisplay 505 which is touched by the user will be set as part of ahandwritten message using a color different than the touch screendisplay background color. Otherwise, the program will await receipt ofpen or stylus input on the touch screen display 505. One skilled in theart recognizes that although the embodiment described in FIG. 7discloses a particular sequence for the decision steps, the handwrittenmessage software program of the present invention may be structured toconsider any number of these and other decision steps, in any order.

Referring now to FIG. 8, an example of the handwritten message which maybe sent in response to the message shown in FIG. 6 is illustrated. Byscripting the words on the touch screen display 505, the user of thePalm™ Handheld Computer is able to send a handwritten reply message, forexample, message 805: “OK—tell him to come to FP this time.”Alternatively, the handwritten reply message could simply have been asymbol such as simply a check mark (i.e., √). In any case, using Dmailaccording to the present invention, the user is able to craft ahandwritten message on the touch screen display 505 of a Palm™ HandheldComputer to quickly, easily, quietly, and unobtrusively communicatewhile in a meeting or traveling in a moving vehicle.

Referring now to FIG. 9, a schematic representation of an exemplarycommunication network according to one embodiment of the presentinvention will now be described. A first handwritten message mobilecommunication device 305A may be coupled to a wireless communicationbase station 905A. The wireless communication base station 905A may becoupled to a public service telephone network (PSTN) 907. Further, thefirst handwritten message mobile communication device 305A may becoupled to another wireless communication base station 905B which may becoupled to the Internet 906. The PSTN 907 and the Internet 906 may becoupled together. A desk top personal computer (PC) 910A with ahandwritten message communication device 305C attached may be coupled tothe Internet 906 and a desk top personal computer (PC) 910B may becoupled to the PSTN 907. Further, a cable or hybrid fiber cable head endhub (HEH) 911 may be coupled to the Internet 906, PSTN 907, and abroadband residential gateway (BRG) 912. In addition, an email server908 and/or a handwritten message server (HW Server) 909 may be coupledto the Internet 906 and the PSTN 907 to support handwritten messagesfrom a handwritten message communication device. A second handwrittenmessage mobile communication device 305B may be coupled to the BRG 912.A third handwritten message mobile communication device 605 may becoupled to wireless base station 905B and/or wireless base station 905A.Further, a facsimile machine 915 may be coupled to PSTN 907.

As noted in FIG. 9, transmissions from the handwritten communicationdevices (305 and 605) may be sent using, for example, industry standardSimple Mail Transfer Protocol (SMTP). Further, messages sent to thehandwritten communication devices (305 and 605) may be sent using, forexample, Post Office Protocol (POP3) or Internet Message Access Protocol(IMAP4).

In operation, a handwritten message image may be sent from any of thehandwritten message communication devices, 305A, 305B, 305C, or 605, toany other communication device such as PC 910A, PC 910B or fax 915 viathe communication system which may include the Internet 906, PSTN 907, awireless network, and/or a satellite network (not shown). Further, ahandwritten message image may be sent from any of the handwrittenmessage communication devices, 305A, 305B, 305C or 605, to each other,with or without being transmitted through a communication network.

One method of operation for sending messages to and from an email serverfor the handwritten communication device (305 and/or 605) will now beexplained. The handwritten communication device (305 and/or 605) mayread incoming messages sent from an email server using industry standardprotocols such as POP3 or IMAP4. These, and all other protocols

mentioned here may be layered on top of the Transmission ControlProtocol (TCP). The handwritten communication device (305 and/or 605)sends outgoing messages to, for example email server 908, using industry

standard SMTP (Simple Mail Transfer Protocol). The bitmaps of ahandwritten message are compressed using a standard such as GIF(Graphics Interface Format) and included within a SMTP message usingMIME (Multipurpose Internet Mail Extensions). SMTP requires that emailbe represented as US-ASCII (American Standard Code for InformationInterchange). MIME does this by encoding the compressed bitmaps inbase64. This increases the file size by 35%.

If the sending and receiving communication devices have differentdisplay sizes, then a means for sizing the handwritten message to thereceiving display size may prove useful. The Palm™ Handheld Computerembodiment presents unique considerations in this respect because of itssmall touch screen display size. The screen size on the Palm™ HandheldComputer is 144×160 pixels. The size of a handwritten image created on atouch screen display size found on the Palm™ Handheld Computer displayedon a normal PC screen (desktop or laptop) is too small to comfortablyread. Therefore, in a preferred embodiment of a handwritten messagecommunication device such as the Palm™ Handheld Computer, the size ofthe image may be increased before it is sent to a PC. Similarly,handwritten messages sent from one handwritten communication device witha larger touch screen display, e.g., say 8½×11 inches, to anotherhandwritten communication device with a smaller touch screen display,e.g., Palm™ Handheld Computer, may need to have the handwritten imagereduced in size and/or allocated to more than one screen (or page) forproper display on the smaller touch screen display. In otherembodiments, the message could be resized by the receiving device or thereceiving and sending devices.

As one approach, for example, a handwritten message image created on aPalm™ Handheld Computer could be double in size by the Palm™ HandheldComputer before sending it to a PC, making it 288×320 pixels. However,this increases the processing time for the compression by a factor of 4and the transmitted file size by a factor of 2.5.

As another approach, a special server, the HW server 909, may beprovided having its own protocol to process outgoing messages from thedevice. The HW Server 909 receives the messages in binary format (savingthe 35%) and in the original 144×160 screen size (saving about 10seconds of processing time on the Palm™ Handheld Computer for thecompression). The HW server 909 does the expansion and MIME encoding andresends the message using SMTP. Since the server will likely have a muchfaster processor than the Palm™ Handheld Computer, this work is donevery quickly on the server. Similarly, the HW server 909 could reduce ahandwritten image size before sending it to a Palm™ Handheld Computer.

The message format between the Palm and the server may be, forexample: - - - destination email address<newline>reply emailaddress<newline>sender's name<newline>subject<newline>compressed imagein binary - - - . Further enhancements, such as authenticating thesender with a password may be included and may be implemented usingapproaches which are well known in the art. (See the IMAP protocol forexample).

Although particular embodiments of the present invention have been shownand described, it will be understood that it is not intended to limitthe invention to the embodiments described herein and it will be obviousto those skilled in the art that various changes and modifications maybe made without departing from the spirit and scope of the presentinvention. Thus, the invention is intended to cover alternatives,modifications, and equivalents, which may be included within the spiritand scope of the invention as defined by the claims.

For example, the present invention may also include coupling a touchscreen display to a PC and providing software on the PC to enable theuser to create and send handwritten message images with a PC. This couldbe a touch screen display the size of a typical sheet of writing paper(e.g., 8½×11 inches), a PDA, or a mobile telephone.

All publications, patents, and patent applications cited herein arehereby incorporated by reference in their entirety for all purposes.

1. An electronic communication device comprising, a touch screen display, and means that enables the electronic communication device to carry out the steps of receiving email that includes a received text message, displaying the received text message for a first user on the touch screen display, the text of the message having been composed by a second user; receiving a handwritten message that was input by the first user using the touch screen display while the received text message was being displayed on the touch screen display, displaying the input handwritten message on the touch screen display in handwritten form at the same time as the received text message is being displayed on the touch screen display, said displaying of the input handwritten message being such that both the received text message and the input handwritten message in handwritten form are displayed on the touch screen display at the same time as one another, in response to the first user thereafter touching a predetermined button area of the touch screen display while both the received text message and the input handwritten message in handwritten form are displayed on the touch screen display at the same time as one another, creating an email message that includes both the received text message and the input handwritten message, and thereafter transmitting the email message to an electronic message address of the second user.
 2. The device of claim 1 wherein the predetermined button area includes the word “reply”.
 3. The device of claim 1 wherein said transmitting comprises transmitting said email message over a communication network.
 4. The device of claim 3 wherein said communication network includes at least one of a wireless communication network, the public switched telephone network, a broadband communication system, and the Internet.
 5. The device of claim 3 wherein said transmitting said email message over a communication network includes transmitting said email message to an electronic mail server connected to said at least one communication network.
 6. The device of claim 3 wherein said transmitting comprises transmitting said email message using an electronic mail protocol.
 7. The device of claim 6 wherein said electronic mail protocol is one of SMTP, POPS and IMAP4. 